Machine Translation in Microblogs

The emergence of social media caused a drastic change in the way information is published. The possibility for people with different backgrounds to publish information has caused non-standard style, formality, content, genre and topics to be present in documents that are published or texted. One such example are posts in microblogs and social networks, such as Twitter, Facebook and Sina Weibo. The people that publish these documents are not professionals, yet the information published can be leveraged for many ends [Han and Baldwin, 2011, Hawn, 2009, Kwak et al., 2010, Sakaki et al., 2010]. However, current NLP tasks, such as Part-of-Speech Tagging perform poorly in the presence of this type of data [Gimpel et al., 2011], since they are trained using traditional features and on existing data. One problem is the lack of annotated datasets in this domain. Another problem is that these models are focused on clean and formal datasets, and make assumptions that do not hold in this domain. One of them is spelling homogeneity, where we assume that there is only one way to spell tomorrow, whereas in microblogs, this word can be abbreviated to tmrw or spelled erroneously as tomorow. In this thesis, we will address the challenge of NLP on the domain of informal online texts, with emphasis on Machine Translation. This thesis makes the following contributions in this respect. (1)We shall present an automatic method to extract such data automatically from microblog posts. Using this in-domain corpora, large improvements can be obtained over systems trained using existing datasets. (2) We also use this corpora to build a microblog normalizer using paraphrasing, which can convert microblog messages into a more standardized text genres, that are more amenable to automatic processing with traditional tools. (3) This will also contribute for better translation quality, and we shall describe methods that will better leverage a microblog normalizer in the MT pipeline. (4)Afterwards, we shall discuss problems with existing evaluation metrics and present metrics that are better suited for this domain. (5) Finally, we shall show applications of the artifacts built in this work on other fields, such as Speech Processing, Natural Language processing and Information Retrieval. We will also show, how the proposed work improves MT in domains other than microblogs.